Unit 27 - Explainable ML Contd

Unit 27: Explainable ML Continued — LIME

Introduction

Building upon our understanding of global surrogate models, this lecture dives deeper into evaluating explainability methods and introduces LIME — one of the most popular techniques for local model interpretability. We explore how to build global surrogates for both classification and regression tasks using decision trees and Lasso, then transition to understanding why individual predictions are made through local approximation.

        Learning Objectives:
        Evaluate explainability methods using R², accuracy, stability, and consistency
Build global surrogates using Decision Trees and Lasso for classification and regression
Understand the LIME algorithm and its local approximation approach
Interpret LIME outputs for both classification and regression examples
Recognize LIME's limitations and best practices for addressing them

    

Theory

1. R-Squared for Measuring Fidelity

R² is a robust measure for evaluating how well a surrogate model replicates a black-box model, especially for regression or when comparing predicted probabilities in classification.

$$R^2 = 1 - \frac{SSE}{SST} = 1 - \frac{\sum_{i=1}^{n}(\hat{y}_{bb}^{(i)} - \hat{y}_{sur}^{(i)})^2}{\sum_{i=1}^{n}(\hat{y}_{bb}^{(i)} - \bar{\hat{y}})^2}$$

Where:

$\hat{y}_{bb}^{(i)}$ = black-box predicted probability for instance $i$
$\hat{y}_{sur}^{(i)}$ = surrogate predicted probability for instance $i$
$\bar{\hat{y}}$ = mean of black-box predictions

Why not just accuracy? Two models can have nearly identical probabilities but different predicted labels after thresholding (e.g., 0.49 vs 0.51 at threshold 0.5). R² captures the continuous similarity, making it superior for fidelity measurement.

2. Fidelity Guidelines

R² (Fidelity)	Interpretation	Use Case
R² > 0.9	Excellent fidelity	Safe for critical decisions
0.7 < R² ≤ 0.9	Good fidelity	Acceptable for most use cases
0.5 < R² ≤ 0.7	Moderate fidelity	Use with caution, validate carefully
R² ≤ 0.5	Poor fidelity	Explanation unreliable, do not use

Rule of Thumb: Never trust an explanation with R² < 0.7.

3. Building Global Surrogates: Classification Example

Using the Adult dataset with a Random Forest black-box model:

# Black-box model
blackbox = Pipeline([
    ("prep", preprocess),
    ("model", RandomForestClassifier(n_estimators=200, random_state=42))
])
blackbox.fit(X_train, y_train)
# Accuracy: 0.861

# Surrogate: Decision Tree (max_depth=3)
surrogate = Pipeline([
    ("prep", preprocess),
    ("model", DecisionTreeClassifier(max_depth=3, random_state=42))
])
surrogate.fit(X_train, blackbox.predict_proba(X_train)[:,1])

# Results:
# Fidelity (Agreement Rate): 0.905
# Fidelity (R²): 0.706
    

4. Simplifying Surrogates with Lasso

Lasso (L1 regularization) is ideal for creating sparse, interpretable surrogate models. By tuning the regularization parameter $\lambda$ (alpha), we control the number of features:

$$\min_{\beta} \sum_{i=1}^{n}(y_i - \beta_0 - \sum_{j=1}^{p}\beta_j x_{ij})^2 + \lambda\sum_{j=1}^{p}|\beta_j|$$

Regularization Strength	Features Selected	Trade-off
High λ	Few (e.g., K=1)	Most interpretable, lowest fidelity
Medium λ	Moderate (e.g., K=5-10)	Good balance (recommended)
Low λ	All (e.g., K=50)	Highest fidelity, hard to interpret

5. Regression Surrogate Example (Car Sales)

Using Gradient Boosting as black-box and comparing Lasso vs. Regression Tree surrogates:

Surrogate	Train Fidelity R²	Test Fidelity R²	Test Accuracy R²	Features Used
Lasso (α=18.4)	0.950	0.947	0.883	24
Regression Tree (depth=3)	0.900	0.909	0.840	4

6. Local Surrogate Models and LIME

LIME (Local Interpretable Model-agnostic Explanations) explains individual predictions by approximating the black-box model locally around a specific instance using an interpretable model.

        Core Idea: Instead of explaining the entire complex decision boundary, LIME fits a simple linear model (dashed line) that's accurate only in the neighborhood of the instance being explained (red cross).
    

Figure: LIME generates perturbed samples around instance X, weights them by proximity, and fits a local linear model (dashed line) to approximate the complex decision boundary.

7. LIME Algorithm Steps

Select instance of interest $x$ for explanation
Perturb dataset by adding noise to features (draw from normal distribution with mean/std from feature)
Get black-box predictions for perturbed points
Weight samples according to proximity to $x$ (e.g., exponential kernel)
Train weighted interpretable model (e.g., linear regression with Lasso) on perturbed data
Interpret local model coefficients as feature contributions

8. LIME Limitations and Best Practices

Key Limitations:

Instability: Explanations for very close points can vary greatly due to random sampling
Sampling variance: Repeating the sampling process can yield different explanations
Manipulation risk: Data scientists can potentially manipulate LIME explanations to hide biases

        Best Practices:
        Run LIME multiple times and check consistency across runs
Combine with global methods (feature importance, global surrogates)
Be especially cautious in high-stakes decisions (medical, legal, financial)
Validate explanations through domain expertise

    

9. Consistency and Stability

Property	Definition	Comparison
Consistency	How much explanations differ between models trained on the same task with similar predictions	Between models (e.g., XGBoost vs. Neural Network)
Stability	How much explanations vary for similar instances in a fixed model	Between similar instances for one model

Interactive Examples

LIME Perturbation Simulator

Simulate how LIME generates perturbed samples around an instance:

Original Instance: Debt Ratio = 0.30, Income = $50K

Black Box Prediction: 0.65 (65% approval probability)

Sample	Debt Ratio	Income ($K)	Distance from X	Weight
1	0.29	51	0.014	0.99
2	0.31	49	0.014	0.99
3	0.35	45	0.071	0.86
4	0.50	30	0.283	0.24

Closer samples receive higher weights, ensuring the local model focuses on the neighborhood of X.

Feature Selection Slider (Lasso)

Adjust the regularization strength to see the trade-off between features and fidelity:

Alpha (λ): Medium (α=10)

Features: 5 | Fidelity R²: 0.85 | Interpretability: Good

Global vs. Local Decision Tree

Figure: Global surrogates provide a single interpretable model for all predictions, while LIME generates a custom local explanation for each individual instance.

Numerical Solutions

Problem 1: Computing Local Fidelity for LIME

Scenario: Loan prediction for an applicant with 30% debt ratio and $50K income. Black box predicts 0.65 approval probability. LIME generates 1000 perturbed samples.

Sample	Debt Ratio	Income ($K)	Black Box Pred	LIME Pred	Error
1	0.29	51	0.67	0.66	0.01
2	0.31	49	0.63	0.64	0.01
...	...	...	...	...	...
1000	0.32	48	0.61	0.62	0.01

Given: Weighted SSE = 0.052, Weighted SST = 0.433. Compute local fidelity R².

Step-by-Step Solution

Step 1: Apply R² Formula

$$R^2 = 1 - \frac{\text{Weighted SSE}}{\text{Weighted SST}} = 1 - \frac{0.052}{0.433}$$

Step 2: Calculate

$$R^2 = 1 - 0.120 = 0.88$$

Step 3: Interpret

R² = 0.88 indicates good local fidelity. The linear approximation is trustworthy in the neighborhood of this instance. However, this explanation should not be generalized beyond similar applicants.

Problem 2: Lasso Regularization Path

A Gradient Boosting model predicts car prices. You fit Lasso surrogates with different α values:

Alpha (α)	Features Selected	Fidelity R²	Actual R²
0.1	58	0.943	0.882
1.0	42	0.940	0.884
10.0	31	0.946	0.887
100.0	14	0.905	0.844
1000.0	5	0.800	0.750

Question: Which α provides the best balance? Justify your answer.

Step-by-Step Solution

Analysis

α = 0.1: 58 features — too many for practical interpretation despite high fidelity.
α = 10.0: 31 features with fidelity 0.946 — good but still many features.
α = 100.0: 14 features with fidelity 0.905 — reasonable balance, but features may still be too many.
α = 1000.0: Only 5 features but fidelity drops to 0.80 — too much information loss.

Recommendation

α = 100.0 or an intermediate value around α = 50-100 provides the best balance. With 10-15 features, the model remains interpretable while maintaining fidelity above the 0.9 threshold. The actual R² of 0.844 is acceptable for most business applications.

Try-It-Yourself Problems

Problem 1: LIME Weighted Regression

Given three perturbed samples around instance X with their black-box predictions and proximity weights:

Sample	Feature 1	Feature 2	BB Pred	Weight
A	2	5	0.8	0.9
B	3	4	0.6	0.7
C	1	6	0.9	0.5

A local linear surrogate predicts: ŷ = 0.2 + 0.1·x₁ + 0.05·x₂. Compute the weighted SSE and comment on whether this is a good local fit.

Step 1: Compute surrogate predictions:

Sample A: ŷ = 0.2 + 0.1(2) + 0.05(5) = 0.2 + 0.2 + 0.25 = 0.65
Sample B: ŷ = 0.2 + 0.1(3) + 0.05(4) = 0.2 + 0.3 + 0.2 = 0.70
Sample C: ŷ = 0.2 + 0.1(1) + 0.05(6) = 0.2 + 0.1 + 0.3 = 0.60

Step 2: Compute weighted SSE:

Weighted SSE = 0.9(0.8-0.65)² + 0.7(0.6-0.70)² + 0.5(0.9-0.60)² = 0.9(0.0225) + 0.7(0.01) + 0.5(0.09) = 0.02025 + 0.007 + 0.045 = 0.07225

Step 3: Compute weighted SST (mean BB pred = 0.767):

Weighted SST = 0.9(0.8-0.767)² + 0.7(0.6-0.767)² + 0.5(0.9-0.767)² = 0.9(0.0011) + 0.7(0.0279) + 0.5(0.0177) ≈ 0.001 + 0.0195 + 0.0089 = 0.0294

Step 4: R² = 1 - (0.07225/0.0294) = -1.46

Conclusion: Negative R²! The local surrogate is worse than simply predicting the mean. This indicates the linear model is inappropriate for this local region, or the perturbation neighborhood is too large.

Problem 2: Surrogate Model Selection

You need to explain a complex Gradient Boosting classifier to:

A regulator who wants to understand overall model behavior
A customer who was denied a loan and wants to know why

Which explainability method(s) would you use for each stakeholder, and why?

1. Regulator (Global Understanding):

Use a Global Surrogate Model (e.g., decision tree with depth 3-5) to capture overall decision rules.
Supplement with feature importance from the Gradient Boosting model.
Report fidelity R² to demonstrate the surrogate's trustworthiness.

2. Customer (Local Explanation):

Use LIME to generate a personalized explanation showing which features contributed to their specific denial.
Present as actionable feedback: "Your application was rejected primarily because your debt-to-income ratio (0.45) was too high."
Include counterfactual: "If your debt ratio were below 0.35, you'd likely be approved."

Problem 3: Consistency Check

You train an XGBoost model and a Neural Network on the same classification task. Both achieve ~87% accuracy. For a test instance, LIME on XGBoost identifies "credit_score" as the top feature with weight +0.15, while LIME on the Neural Network identifies "income" as top with weight +0.18 for the same instance. What does this suggest about consistency, and what should you do?

Analysis:

Low consistency is indicated because two models with similar accuracy give different explanations for the same instance.
This could mean: (a) the models learned genuinely different decision boundaries, (b) LIME's sampling variance is high, or (c) both features are correlated and either could drive the prediction.

Actions:

Run LIME multiple times on both models to check if the difference persists (stability check).
Examine the correlation between credit_score and income — if highly correlated, either could be the true driver.
Use SHAP as a complementary method to verify feature importance.
Investigate if one model is using a spurious correlation (e.g., credit_score proxying for income due to data bias).

Interactive Quiz

Question 1: Why is R² preferred over accuracy for measuring surrogate fidelity in classification?

R² is always higher than accuracy

R² works on probabilities without thresholding, capturing continuous similarity

Accuracy is not defined for classification tasks

R² requires less computation

Question 2: What is the primary purpose of LIME?

To replace black-box models with interpretable ones globally

To explain individual predictions through local approximation

To improve the accuracy of black-box models

To reduce the dimensionality of datasets

Question 3: A Lasso surrogate with high regularization (large α) will:

Include all features with high fidelity

Select fewer features, trading fidelity for interpretability

Always achieve R² > 0.95

Become identical to the black-box model

Question 4: Which of the following is a known limitation of LIME?

It only works for neural networks

Explanations can be unstable across repeated runs

It requires access to model gradients

It cannot handle categorical features

Question 5: Stability in explainability refers to:

How similar explanations are between different model types

How much explanations vary for similar instances in the same model

The computational speed of generating explanations

Whether the explanation matches ground truth labels

Key Takeaways

R² is the preferred fidelity metric for regression surrogates and classification probability surrogates because it captures continuous similarity without thresholding.
Never trust explanations with R² < 0.7. Aim for R² > 0.9 for critical decisions.
Lasso regularization provides a principled way to balance interpretability and fidelity by controlling the number of features in a linear surrogate.
Decision tree surrogates (depth 3-5) offer intuitive rule-based explanations and are excellent for communicating with non-technical stakeholders.
LIME provides local explanations by perturbing instances, weighting by proximity, and fitting a simple linear model in the local neighborhood.
LIME has limitations: instability, sampling variance, and potential for manipulation. Always run multiple times and combine with global methods.
Consistency compares explanations across models; stability compares explanations across similar instances for one model.

Common Pitfalls

Using accuracy instead of R² for fidelity: Thresholding probabilities can mask true model agreement. Always use R² for probability-based fidelity.
Over-regularizing Lasso: Setting α too high yields an overly simple model with poor fidelity. Use cross-validation to find the optimal balance.
Trusting a single LIME run: Due to random perturbation sampling, run LIME multiple times and check for consistent feature rankings.
Generalizing local explanations: LIME explains one instance's neighborhood. Do not assume the same features are globally important.
Ignoring surrogate complexity: A decision tree with 20 levels is not interpretable. Limit depth to 3-5 for human consumption.
Confusing consistency with stability: Consistency is across models; stability is across instances. Both are necessary for trustworthy explanations.

Resources

📚 Interpreting Machine Learning Models by Nandi and Pal
📚 Interpretable Machine Learning by Christoph Molnar
🔗 LIME Official GitHub Repository
🔗 LIME Chapter (Molnar's Book)
🔗 Scikit-learn Lasso Documentation
💻 Python Libraries: lime, sklearn.linear_model.Lasso, sklearn.tree.DecisionTreeRegressor/Classifier